Quantifying Inductive Bias: AI Learning Algorithms and Valiant's Learning Framework

We show that the notion of inductive bias in concept learning can be quantified in a way that directl_v relates to learning performance in the framework recently introduced by Valiant. Our measure of bias is based on the growth function introduced by Vapnik and Chervonenkis, and on the Vapnik-Chervonenkis dimension. We measure some common language biases, including restriction to conjunctive concepts, conjunctive concepts with internal disjunction, k-DNF and k-CNF concepts. We also measure certain types of bias that result from a preference for simpler hypotheses. Using these bias measurements we analyze the performance o f the classical learning algorithm ]or ~onjunctive concepts from the perspective of Valiant's learning framework. We then augment this algorithm with a hypothesis simplification routine that uses a greed~v heuristic and show how this improves learning performance on simpler target concepts. Improved learning algorithms are also developed [~r conjunctive concepts with internal disjunction, k-DNF and k-CNF concepts. We show that all our algorithms are within a logarithmic ]hctor of optimal in terms of the namber of examples th O' require to achieve a given level o f learning performance in the Valiant .[?amework. Our results hold .[?~r arbitrary attribute-based instance spaces defined by either tree-structured or linear attributes.